{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第09课：一网打尽神经序列模型之 RNN 及其变种 LSTM、GRU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当人工神经网络从浅层发展到深层；从全连接到卷积神经网络。在此过程中，人类在图片分类、语音识别等方面都取得了非常好的结果，那么我们为什么还需要循环神经网络呢？\n",
    "上面提到的这些网络结构的层与层之间是全连接或部分连接的，但在每层之间的节点是无连接的，这样的网络结构并不能很好的处理序列数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram 模型是一种语言模型（Language Model，LM），是一个基于概率的判别模型，它的输入是一句话（词的顺序序列），输出是这句话的概率，即这些词的联合概率（Joint Probability）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vec = CountVectorizer(\n",
    "    analyzer='word', # tokenise by character ngrams\n",
    "    ngram_range=(1,4),  # use ngrams of size 1 and 2\n",
    "    max_features=20000,  # keep the most common 1000 ngrams\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram 模型，在自然语言处理中主要应用在如词性标注、垃圾短信分类、分词器、机器翻译和语音识别、语音识别等领域。\n",
    "然而 N-gram 模型并不是完美的，它存在如下优缺点：\n",
    "\n",
    "优点：包含了前 N-1 个词所能提供的全部信息，这些词对于当前词的出现概率具有很强的约束力；\n",
    "\n",
    "缺点：需要很大规模的训练文本来确定模型的参数，当 N 很大时，模型的参数空间过大。所以常见的 N 值一般为1，2，3等。还有因数据稀疏而导致的数据平滑问题，解决方法主要是拉普拉斯平滑和内插与回溯。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据 N-gram 的优缺点，它的进化版 NNLM（Neural Network based Language Model）\n",
    "\n",
    "由四层组成，输入层、嵌入层、隐层和输出层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NNLM 接收的输入是长度为 N 的词序列，输出是下一个词的类别。首先，输入是词序列的 index 序列，例如词“我”在字典（大小为|V|）中的 index 是10，词“是”的 index 是23， “小明”的 index 是65，则句子“我是小明”的 index 序列就是 10、 23、65。嵌入层（Embedding）是一个大小为 |V|×K 的矩阵，从中取出第10、23、65行向量拼成 3×K 的矩阵就是 Embedding 层的输出了。隐层接受拼接后的 Embedding 层输出作为输入，以 tanh 为激活函数，最后送入带 softmax 的输出层，输出概率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN 为序列数据而生\n",
    "RNN 称为循环神经网路，因为这种网络有“记忆性”，主要应用在自然语言处理（NLP）和语音领域。RNN 具体的表现形式为网络会对前面的信息进行记忆并应用于当前输出的计算中，即隐藏层之间的节点不再无连接而是有连接的，并且隐藏层的输入不仅包括输入层的输出还包括上一时刻隐藏层的输出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "理论上，RNN 能够对任何长度的序列数据进行处理，但由于该网络结构存在“梯度消失”问题，所以在实际应用中，解决梯度消失的方法有：梯度裁剪（Clipping Gradient）和 LSTM（Long Short-Term Memory）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTM 通过三个“门”结构来控制不同时刻的状态和输出。所谓的“门”结构就是使用了 Sigmoid 激活函数的全连接神经网络和一个按位做乘法的操作，Sigmoid 激活函数会输出一个0~1之间的数值，这个数值代表当前有多少信息能通过“门”，0表示任何信息都无法通过，1表示全部信息都可以通过。其中，“遗忘门”和“输入门”是 LSTM 单元结构的核心。下面我们来详细分析下三种“门”结构。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "遗忘门，用来让 LSTM“忘记”之前没有用的信息。它会根据当前时刻节点的输入  Xt上一时刻节点的状态 Ct−1和上一时刻节点的输出 h_{t-1}来决定哪些信息将被遗忘。\n",
    "\n",
    "输入门，LSTM 来决定当前输入数据中哪些信息将被留下来。在 LSTM 使用遗忘门“忘记”部分信息后需要从当前的输入留下最新的记忆。输入门会根据当前时刻节点的输入 X_t上一时刻节点的状态 C_{t-1}和上一时刻节点的输出 h_{t-1}来决定哪些信息将进入当前时刻节点的状态 C_t模型需要记忆这个最新的信息。\n",
    "\n",
    "输出门，LSTM 在得到最新节点状态 C_t后，结合上一时刻节点的输出 h_{t-1}和当前时刻节点的输入 X_t来决定当前时刻节点的输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#引入包\n",
    "import random\n",
    "import jieba\n",
    "import pandas as pd\n",
    "#加载停用词\n",
    "stopwords=pd.read_csv('stopwords.txt',index_col=False,quoting=3,sep=\"\\t\",names=['stopword'], encoding='utf-8')\n",
    "stopwords=stopwords['stopword'].values\n",
    "\n",
    "#加载语料\n",
    "laogong_df = pd.read_csv('../data/06/beilaogongda.csv', encoding='utf-8', sep=',')\n",
    "laopo_df = pd.read_csv('../data/06/beilaogongda.csv', encoding='utf-8', sep=',')\n",
    "erzi_df = pd.read_csv('../data/06/beierzida.csv', encoding='utf-8', sep=',')\n",
    "nver_df = pd.read_csv('../data/06/beinverda.csv', encoding='utf-8', sep=',')\n",
    "#删除语料的nan行\n",
    "laogong_df.dropna(inplace=True)\n",
    "laopo_df.dropna(inplace=True)\n",
    "erzi_df.dropna(inplace=True)\n",
    "nver_df.dropna(inplace=True)\n",
    "#转换\n",
    "laogong = laogong_df.segment.values.tolist()\n",
    "laopo = laopo_df.segment.values.tolist()\n",
    "erzi = erzi_df.segment.values.tolist()\n",
    "nver = nver_df.segment.values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /var/folders/yw/k7z_d_3567g16ss9plk47x9w0000gn/T/jieba.cache\n",
      "Loading model cost 1.719 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    }
   ],
   "source": [
    "#定义分词和打标签函数preprocess_text\n",
    "#参数content_lines即为上面转换的list\n",
    "#参数sentences是定义的空list，用来储存打标签之后的数据\n",
    "#参数category 是类型标签\n",
    "def preprocess_text(content_lines, sentences, category):\n",
    "    for line in content_lines:\n",
    "        try:\n",
    "            segs=jieba.lcut(line)\n",
    "            segs = [v for v in segs if not str(v).isdigit()]#去数字\n",
    "            segs = list(filter(lambda x:x.strip(), segs)) #去左右空格\n",
    "            segs = list(filter(lambda x:len(x)>1, segs))#长度为1的字符\n",
    "            segs = list(filter(lambda x:x not in stopwords, segs)) #去掉停用词\n",
    "            sentences.append((\" \".join(segs), category))# 打标签\n",
    "        except Exception:\n",
    "            print(line)\n",
    "            continue \n",
    "\n",
    "#调用函数、生成训练数据\n",
    "sentences = []\n",
    "preprocess_text(laogong, sentences,0)\n",
    "preprocess_text(laopo, sentences, 1)\n",
    "preprocess_text(erzi, sentences, 2)\n",
    "preprocess_text(nver, sentences, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "报警 人称 儿子 人无事 民警 到场 2\n",
      "老公 人伤 无需 持械 民警 到场 0\n",
      "报警 儿子 打人 对象 无需 民警 到场 民警 携带 防护 装备 处置 2\n",
      "报警 儿子 民警 到场 民警 携带 防护 装备 2\n",
      "报警 人称 丈夫 一人伤 无需 救护 民警 到场 1\n",
      "刚刚 儿子 持械 无需 救护 2\n",
      "老公 持械 无需 救护 民警 到场 1\n",
      "报警 儿子 持械 人伤 无需 救护车 民警 到场 民警 携带 防护 设备 2\n",
      "报警 女儿 民警 到场 3\n",
      "报警 人称 儿子 无需 救护 民警 到场 2\n"
     ]
    }
   ],
   "source": [
    "#打散数据，生成更可靠的训练集\n",
    "random.shuffle(sentences)\n",
    "\n",
    "#控制台输出前10条数据，观察一下\n",
    "for sentence in sentences[:10]:\n",
    "    print(sentence[0], sentence[1])\n",
    "#所有特征和对应标签\n",
    "all_texts = [ sentence[0] for sentence in sentences]\n",
    "all_labels = [ sentence[1] for sentence in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/Users/zhangjianfeng/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "#引入需要的模块\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils import to_categorical\n",
    "from keras.layers import Dense, Input, Flatten, Dropout\n",
    "from keras.layers import LSTM, Embedding,GRU\n",
    "from keras.models import Sequential\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#预定义变量\n",
    "MAX_SEQUENCE_LENGTH = 100    #最大序列长度\n",
    "EMBEDDING_DIM = 200    #embdding 维度\n",
    "VALIDATION_SPLIT = 0.16    #验证集比例\n",
    "TEST_SPLIT = 0.2    #测试集比例\n",
    "#keras的sequence模块文本序列填充\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(all_texts)\n",
    "sequences = tokenizer.texts_to_sequences(all_texts)\n",
    "word_index = tokenizer.word_index\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 391 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "print('Found %s unique tokens.' % len(word_index))\n",
    "data = pad_sequences(sequences, maxlen=100)\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (1722, 100)\n",
      "Shape of label tensor: (1722, 4)\n"
     ]
    }
   ],
   "source": [
    "labels = to_categorical(np.asarray(all_labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#数据切分\n",
    "p1 = int(len(data)*(1-VALIDATION_SPLIT-TEST_SPLIT))\n",
    "p2 = int(len(data)*(1-TEST_SPLIT))\n",
    "x_train = data[:p1]\n",
    "y_train = labels[:p1]\n",
    "x_val = data[p1:p2]\n",
    "y_val = labels[p1:p2]\n",
    "x_test = data[p2:]\n",
    "y_test = labels[p2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 100, 200)          78400     \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 200)               320800    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                12864     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 4)                 260       \n",
      "=================================================================\n",
      "Total params: 412,324\n",
      "Trainable params: 412,324\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "['loss', 'acc']\n",
      "Train on 1102 samples, validate on 275 samples\n",
      "Epoch 1/10\n",
      "1102/1102 [==============================] - 8s 7ms/step - loss: 1.3569 - acc: 0.3158 - val_loss: 1.2525 - val_acc: 0.5491\n",
      "Epoch 2/10\n",
      "1102/1102 [==============================] - 4s 4ms/step - loss: 1.1244 - acc: 0.5907 - val_loss: 1.2300 - val_acc: 0.4909\n",
      "Epoch 3/10\n",
      "1102/1102 [==============================] - 4s 4ms/step - loss: 0.7854 - acc: 0.6760 - val_loss: 0.6716 - val_acc: 0.7200\n",
      "Epoch 4/10\n",
      "1102/1102 [==============================] - 5s 4ms/step - loss: 0.5188 - acc: 0.7405 - val_loss: 0.3753 - val_acc: 0.7600\n",
      "Epoch 5/10\n",
      "1102/1102 [==============================] - 5s 4ms/step - loss: 0.4353 - acc: 0.7441 - val_loss: 0.3504 - val_acc: 0.7455\n",
      "Epoch 6/10\n",
      "1102/1102 [==============================] - 5s 4ms/step - loss: 0.3735 - acc: 0.7350 - val_loss: 0.3610 - val_acc: 0.7636\n",
      "Epoch 7/10\n",
      "1102/1102 [==============================] - 4s 4ms/step - loss: 0.3601 - acc: 0.7695 - val_loss: 0.3362 - val_acc: 0.7418\n",
      "Epoch 8/10\n",
      "1102/1102 [==============================] - 4s 4ms/step - loss: 0.3779 - acc: 0.7414 - val_loss: 0.3466 - val_acc: 0.7673\n",
      "Epoch 9/10\n",
      "1102/1102 [==============================] - 5s 4ms/step - loss: 0.3629 - acc: 0.7632 - val_loss: 0.3421 - val_acc: 0.7636\n",
      "Epoch 10/10\n",
      "1102/1102 [==============================] - 5s 4ms/step - loss: 0.3611 - acc: 0.7604 - val_loss: 0.3373 - val_acc: 0.7564\n",
      "345/345 [==============================] - 0s 1ms/step\n",
      "[0.38240776493929435, 0.75072463975436443]\n"
     ]
    }
   ],
   "source": [
    "#LSTM训练模型\n",
    "model = Sequential()\n",
    "model.add(Embedding(len(word_index) + 1, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH))\n",
    "model.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(64, activation='relu'))\n",
    "model.add(Dense(labels.shape[1], activation='softmax'))\n",
    "model.summary()\n",
    "#模型编译\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "print(model.metrics_names)\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=10, batch_size=128)\n",
    "model.save('lstm.h5')\n",
    "#模型评估\n",
    "print(model.evaluate(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 100, 200)          78400     \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 200)               240600    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 200)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 64)                12864     \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 4)                 260       \n",
      "=================================================================\n",
      "Total params: 332,124\n",
      "Trainable params: 332,124\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "['loss', 'acc']\n",
      "Train on 1102 samples, validate on 275 samples\n",
      "Epoch 1/10\n",
      "1102/1102 [==============================] - 4s 3ms/step - loss: 1.3291 - acc: 0.4065 - val_loss: 1.2431 - val_acc: 0.3564\n",
      "Epoch 2/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.9976 - acc: 0.6488 - val_loss: 0.9680 - val_acc: 0.6327\n",
      "Epoch 3/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.7036 - acc: 0.7396 - val_loss: 0.5142 - val_acc: 0.7418\n",
      "Epoch 4/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.4670 - acc: 0.7468 - val_loss: 0.3847 - val_acc: 0.7345\n",
      "Epoch 5/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3831 - acc: 0.7514 - val_loss: 0.3435 - val_acc: 0.7127\n",
      "Epoch 6/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3709 - acc: 0.7350 - val_loss: 0.3385 - val_acc: 0.7636\n",
      "Epoch 7/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3602 - acc: 0.7514 - val_loss: 0.3373 - val_acc: 0.7018\n",
      "Epoch 8/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3585 - acc: 0.7595 - val_loss: 0.3430 - val_acc: 0.7345\n",
      "Epoch 9/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3568 - acc: 0.7604 - val_loss: 0.3328 - val_acc: 0.7527\n",
      "Epoch 10/10\n",
      "1102/1102 [==============================] - 3s 3ms/step - loss: 0.3556 - acc: 0.7713 - val_loss: 0.3671 - val_acc: 0.7636\n",
      "345/345 [==============================] - 0s 1ms/step\n",
      "[0.41007884688999341, 0.71884057832800818]\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(len(word_index) + 1, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH))\n",
    "model.add(GRU(200, dropout=0.2, recurrent_dropout=0.2))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(64, activation='relu'))\n",
    "model.add(Dense(labels.shape[1], activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "print(model.metrics_names)\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=10, batch_size=128)\n",
    "model.save('lstm.h5')\n",
    "\n",
    "print(model.evaluate(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
